System and method for measuring tracker system accuracy

ABSTRACT

The present invention relates to a simple and effective system and method for measuring camera based tracker system accuracy, especially for a helmet-mounted tracker system, utilizing Coordinate Measuring Machine (CMM). The method comprises the steps of; computing spatial relation between tracked object and calibration pattern using CMM; computing relation between reference camera and tracker camera; computing relation between reference camera and calibration pattern; computing ground truth relation between tracker camera and tracked object; obtaining actual tracker system results; comparing these results with the ground truth relations and finding accuracy of the tracker system; recording accuracy results; testing if the accuracy results is a new calculation required. The system comprises; a reference camera; a calibration pattern visible by reference camera; a camera spatial relation computation unit; a relative spatial relation computation unit a memory unit; a spatial relation comparison unit.

FIELD OF THE INVENTION

The present invention relates to the field of measuring, testing andmethodologies to determine the accuracy of systems optically trackingobjects, especially helmet-mounted tracker systems.

BACKGROUND OF THE INVENTION

It is known that there are methods and models to track a threedimensional object in an environment and compute its position andorientation with respect to a predetermined coordinate system. Thesekinds of tracker systems are used for example in aircrafts to determinethe orientation of head of the pilot. Once the orientation is acquiredwith respect to the coordinate system of say the display devices, thenit is possible to generate graphics on these accordingly. There aredifferent methods to track an object in the scene using magnetic,mechanical or optical means. Currently, the spatial relations of objectsmay also be determined using magnetic sensors or laser beams but thisinvention relates specifically to systems using camera-based (day-tv,thermal, IR, Time of Flight etc.) trackers.

In one of the optical camera-based systems the pilot wears a helmet withmarks or patterns on and at least one tracker camera determines thehelmet's position and orientation using coordinate transformationcalculations based on these patterns. Computing spatial relation betweenan object having a tracking pattern, and a camera is therefore, wellknown in the state of the art. Throughout the document, whenever aspatial relation is mentioned, it should be understood that the relationbetween an entity's predetermined reference system with respect to theother's is meant. This reference system is generally based on therespective pattern of an object under consideration. Since the positionof the tracker camera with respect to the other systems is known (or canbe calculated or measured), it is also possible to compute the helmet'sspatial relation with the tracker camera's sensor and then with othersystems. Computing spatial relation between different identities, giventheir spatial relation with respect to a known reference frame is alsopossible with related vector translations and matrix calculations. Inthe same manner, similar concepts can also be applied to robotic systemsto determine the spatial relation of various arms of the roboticproduct. Nevertheless, a tracker system using a tracker camera tracks anobject (say tracked object) and calculates its position and orientationwith respect to a known reference frame. Then this relative relation ofthe tracked object is used for many different purposes. In this context,“tracked object” means an object having a tracking pattern and beingtracked by a tracker system. It may be either a helmet as in ahelmet-mounted tracker system or any other object.

The patterns used in camera-based tracker systems are either graphical(generally black and white) patterns (passive marker) tracked by visiblelight cameras or arrays of infrared LEDs (active marker) tracked byinfrared cameras. Other arrangements are also possible but the mostconvenient among them is the one with the infrared LEDs since thesesystems can work under inappropriate lighting conditions. There are alsosome problems related to these tracker systems such as calibration oraccuracy determination. Calibration and testing of such systems aregenerally cumbersome, difficult and require complicated equipment.Furthermore, sometimes instead of calibrating the tracker system, it isessential to determine the system's current state's accuracy,

In a currently used calibration method, a laser tracker is used todetermine the orientation of different objects relative to a referencepoint in the scene and the system under consideration is arranged sothat it is consistent. Although this method results in an accuratecalibration, it uses a laser tracker which is an expensive solution whenit is only required to determine the current accuracy of the systemunder consideration.

Another method which is currently used to calibrate a head trackersystem uses specially adapted mechanisms to change the positions of themarkers on the scene and then configures the tracker system using thesedata, generating a coordinate system. Again this method is not practicaland uses special equipment which makes the process complicated. When itis only necessary to determine the current accuracy of the system, theerror of the current system must be deduced.

The current methods are not offering a simple and efficient way of onlymeasuring a tracker system's accuracy while tracking an object. Toprovide a solution to this problem, a new methodology should beintroduced which uses simpler tools and steps.

The Chinese patent document CN101524842, an application in the state ofthe art, discloses a method calibrating a robot, taking an initial testpoint as a reference, and establishing a test coordinate system with alaser tracker, tracking curve arc with the laser tracker, and recordingposition coordinates measured from clockwise direction andcounter-clockwise direction.

The Japanese patent document JP2009085806, an application in the stateof the art, discloses a method for accurately setting a cameracoordinate system and calibrating the head tracker system by moving theoptical marker with a stage mechanism and taking measurements.

SUMMARY OF THE INVENTION

An objective of the present invention is to provide a simple andeffective methodology to measure a camera based tracker system'saccuracy.

DETAILED DESCRIPTION OF THE INVENTION

A system and method realized to fulfil the objective of the presentinvention is illustrated in the accompanying figures, in which:

FIG. 1 is the schematic view of the preferred embodiment system.

FIG. 2 is the flowchart of the preferred method of the presentinvention.

The components illustrated in the figures are individually referencedwhere the numbers and letters refer to the following:

-   1. System for measuring tracker accuracy-   2. Reference camera-   3. Calibration pattern-   4. Camera spatial relation computation unit-   5. Relative spatial relation computation unit-   6. Spatial relation comparison unit-   7. Memory unit-   O. Tracked object-   S. Tracker system-   T. Tracker camera-   100. Method for measuring tracker system accuracy

A system for measuring tracker system accuracy (1) fundamentallycomprises;

-   -   at least one reference camera (2) configured to acquire at least        one electronic pixel image of at least one tracked object (0)        having a tracking pattern and being tracked by a tracker camera        (T),    -   at least one calibration pattern (3) which is visible by        reference camera (2),    -   at least one camera spatial relation computation unit (4) which        is connected to at least one camera and configured to compute        spatial relation between an object having a tracking pattern and        a camera using the object's electronic pixel image acquired by        these cameras,    -   at least one relative spatial relation computation unit (5)        which is connected to the camera spatial relation computation        unit (4) and configured to compute spatial relation between at        least two identities, given the spatial relation of all these        identities with respect to a reference frame,    -   at least one memory unit (7) configured to store at least one        accuracy value,    -   at least one spatial relation comparison unit (6) which is        connected to camera spatial relative spatial relation        computation unit (5), tracker system (S) and memory unit (7),        and configured to compare the tracker system's (S) tracker        camera (T)—tracked object (0) relation results with a given        ground truth, finding an accuracy value.

In a preferred embodiment of the present invention, reference camera (2)is a camera of the same type as the tracker camera (T) of the trackersystem (S). For example if the tracker camera (T) is an infrared visioncamera tracking a LED pattern, the reference camera (2) is also aninfrared vision camera.

In a preferred embodiment of the present invention, calibration pattern(3) is also visible by the tracker camera (T) to be able to determinereference camera's (2) spatial relation with respect to the trackercamera (T), without using any external tool. This is realized bycalculating spatial relation of the reference camera (2) with respect tothe calibration pattern (3) and calculating spatial relation of thetracker camera (T) with respect to the calibration pattern using cameraspatial relation computation unit (4). Then using the relative spatialrelation computation unit (5), reference camera's (2) spatial relationwith respect to the tracker camera (T) is established with the requiredtransformation calculations. Therefore, in this embodiment, referencecamera (2) and tracker camera (T) of the tracker system (S), for whichthe accuracy measurement will be made, are connected to the cameraspatial relation computation unit (4). This means that, the cameraspatial relation computation unit (4) should be compatible andconnectible with tracker camera (T) under consideration and isconfigured to determine spatial relation of the tracker camera (T) andreference camera (2) with respect to the calibration pattern (3).Relative spatial relation computation unit (5) on the other hand, isconfigured to determine the relation of the reference camera (2) to thetracker camera (T) using their relation information with the calibrationpattern (3).

Preferentially, calibration pattern (3) is a checker pattern or apattern composed of ellipses since corners or blobs can be located veryaccurately in the captured image, knowing that the pattern is on asmooth planer surface. Also, it is mountable on the tracked object (O)with a known spatial relation with the object (O) which was previouslymeasured by CMM (coordinate measuring machine), laser scanning systemsor any other accurate method. Such an embodiment makes the spatialrelation between tracked object (O) and calibration pattern (3) readilyknown as the calibration pattern (3) is mounted on the object (O). Inanother embodiment, similarly, reference camera (2) is mounted on aplace with a known relation with the tracker camera (T) and relationbetween reference camera (2) and tracker camera (T) becomes apparentwithout any calculations.

Knowing the spatial relation between tracked object (O) and calibrationpattern (3); reference camera (2) and tracker camera (T); and referencecamera (2) and calibration pattern (3); it is possible to determinetracked object's (O) relation with the tracker camera (T) from thereference camera (2) viewpoint, using relative spatial relationcomputation unit (5). This relation data actually provides a groundtruth for the object's (O) relation with the tracker camera (T) and canbe safely used to determine tracker system (S) accuracy.

In another preferred embodiment, spatial relation comparison unit (6) isconfigured to compare the tracker system's (S) spatial relation resultswith the ground truth calculated by relative spatial relationcomputation unit (5) and record the accuracy result in memory unit (7).

In yet another preferred embodiment of the invention, both calibrationpattern (3) and tracked object (O) are visible by the tracker camera (T)and reference camera (2) is not necessary. Reference camera (2) is onlyincluded in the other embodiment to calculate accuracy for a differentpose of the tracked object (O). If tracker Camera (T) can see thecalibration pattern(3) and tracked object (O) for all locations of thetracked object (O), system can work by using the tracker Camera(T) as aReference Camera. In this situation, there is no need to calculate therelation between reference camera (2) and tracker Camera(T) and stillthe same calculations are possible. In this case, relative spatialrelation computation unit (5) is configured to determine trackedobject's (O) ground truth relation with the tracker camera (T) from thetracker camera (T) viewpoint, using the spatial relation between trackedobject (O) and calibration pattern (3); and tracker camera (T) andcalibration pattern (3).

A method for measuring tracker system accuracy (100) fundamentallycomprises the following steps,

-   -   computing spatial relation between a tracked object (O) and a        calibration pattern (3) (101),    -   computing spatial relation between a reference camera (2) and a        tracker camera (T) (102),    -   computing spatial relation between a reference camera (2) and a        calibration pattern (3) using camera spatial relation        computation unit (4) (103),    -   computing a ground truth spatial relation between the tracker        camera (T) and the tracked object (O) with the data from        previous steps (104) and using relative spatial relation        computation unit (5),    -   obtaining tracker system (S) results giving the spatial relation        between the tracker camera (T) and the tracked object (O) (105),    -   comparing tracker system (S) results with the calculated ground        truth relations and computing accuracy of the tracker system (S)        using spatial relation comparison unit (6) (106),    -   recording accuracy results for the current pose of the tracked        object (O) to memory unit (7) (107), and    -   is a new calculation required? (108).

First, spatial relation between a tracked object (O) and a calibrationpattern (3) (101) is calculated by any known means of calculation. Forexample by a CMM, laser system. In a preferred configuration, thesecomputations are done using CMM measurements. Spatial relation between areference camera (2) and a tracker camera (T) (102) is also calculatedby any known means of calculation. Camera spatial relation computationunit (4) and relative spatial relation computation unit (5) is used inpreferred configuration.

Then the spatial relation between a reference camera (2) and acalibration pattern (3) is calculated using camera spatial relationcomputation unit (4) (103). In the following step, tracked object's (O)relation with the tracker camera (T) from the reference camera (2)viewpoint is determined using relative spatial relation computation unit(5) using the spatial relation between tracked object (O) andcalibration pattern (3); reference camera (2) and tracker camera (T);and reference camera (2) and calibration pattern (3). All these threerelations were found in steps (101), (102) and (103). The relationbetween tracked object (O) and tracker camera (T) from the referencecamera (2) viewpoint actually provides a ground truth and can be safelyused to determine tracker system (S) accuracy.

After step (104), tracker system (S) results, giving the spatialrelation between the tracker camera (T) and the tracked object (O), aredirectly received from the tracker system (S) as is (105). These datawill be the original data to measure accuracy and these are comparedwith the ground truth relations found in the previous steps and accuracyof the tracker system (S) is calculated using spatial relationcomparison unit (6) (106). Finally, the accuracy results for the currentpose of the tracked object (O) are recorded to memory unit (7) (107).

In order to record accuracy measurements for different poses of thetracked object (O), a new calculation required? (108) check is doneafter step (107) and all the calculations are done and recorded for thenew pose of the object (O) starting from step (101). This may berequired since the accuracy of the tracker system (S) may be differentfor different poses of the object (O) and different accuracy values fordifferent poses may be required. If no new calculation is required, thenthe method ends. Additionally, step (101) becomes unnecessary whenspatial relation between tracked object (O) and a calibration pattern(3) is previously known as in the case where calibration pattern (3) ismounted on the tracked object (O) and a CMM measurement is made for thatreference camera (2) position. Step (102) also becomes unnecessary whenspatial relation between reference camera (2) and tracker camera (T) ispreviously known and they are stationary. Therefore steps (101) and(102) are practiced only when the respective relation is not previouslyknown. For example for a specific tracker system (S), a setting with acalibration pattern (3) mounted on a tracked object (O) and with a fixedreference camera (2), steps (101) and (102) will only be performed once.

The method (100) together with the system (1) can simply and effectivelymeasure a camera based tracker system's accuracy by calculating a groundtruth relation between the tracked object (O) and the tracker camera(T).

Within the scope of these basic concepts, it is possible to develop awide variety of embodiments of the inventive “system and method formeasuring tracker system accuracy” (1), (100). The invention cannot belimited to the examples described herein; it is essentially according tothe claims.

1. A system for measuring tracker system accuracy that exploits CMM incomputing relative pose of calibration patterns with tracked objectscomprising; at least one camera spatial relation computation unit whichis connected to at least one camera and configured to compute spatialrelation between an object having a tracking pattern and a camera usingthe object's electronic pixel image acquired by these cameras, at leastone spatial relation computation unit, that computes the pose of thetracked object with respect to a pre-defined coordinate system usingcamera spatial relation computation unit, at least one reference cameraconfigured to acquire at least one electronic pixel image of acalibration pattern mounted on at least one tracked object and onetracked object, being tracked by a tracker camera, at least onecalibration pattern which is visible by reference camera, for which thespatial relation with respect to tracked object is measured with a CMM,at least one ground truth pose value computed via the calibrationpattern and the aforementioned CMM measurements, at least one memoryunit configured to store at least one ground truth pose value andaccuracy value, at least one spatial relation comparison unit configuredto compare the tracker system's results with a given ground truth,finding an accuracy value.
 2. The system for measuring tracker systemaccuracy according to claim 1, wherein a calibration pattern which isvisible by the tracker camera and for which CMM is utilized to measurethe pose with respect to the tracked object.
 3. The system for measuringtracker system accuracy according to claims 1, wherein the relativespatial relation computation unit which is configured to determinetracked object's ground truth relation with the tracker camera from thereference camera viewpoint, using the spatial relation between trackedobject and the calibration pattern; the reference camera and the trackercamera; and the reference camera and the calibration pattern.
 4. Thesystem for measuring tracker system accuracy according to claim 3,wherein the spatial relation comparison unit which is configured tocompare the tracker system's spatial relation results with the groundtruth calculated by relative spatial relation computation unit.
 5. Thesystem for measuring tracker system accuracy according to claim 1,wherein the calibration pattern which is a pattern composed of ellipses.6. The system for measuring tracker system accuracy as in claims 1,wherein the calibration pattern which is mounted on the tracked objectwith a known previously measured spatial relation with the trackedobject, measured by the CMM.
 7. The method for measuring tracker systemaccuracy, which comprises the steps of; computing spatial relationbetween a tracked object and a calibration pattern using a CMM,computing spatial relation between a reference camera and a trackercamera, computing spatial relation between the reference camera and thecalibration pattern using a camera spatial relation computation unit,computing a ground truth spatial relation between the tracker camera andthe tracked object with the data from previous steps and using therelative spatial relation computation unit, obtaining tracker systemresults giving the spatial relation between the tracker camera and thetracked object, comparing tracker system results with the calculatedground truth relations and computing accuracy of the tracker systemusing spatial relation comparison unit, recording accuracy results forthe current pose of the tracked object to memory unit, and testing ifthe accuracy results is a new calculation required.
 8. The system formeasuring tracker system accuracy according to claim 2, wherein therelative spatial relation computation unit which is configured todetermine tracked object's ground truth relation with the tracker camerafrom the reference camera viewpoint, using the spatial relation betweentracked object and the calibration pattern; the reference camera and thetracker camera; and the reference camera and the calibration pattern. 9.The system for measuring tracker system accuracy according to claim 8,wherein the spatial relation comparison unit which is configured tocompare the tracker system's spatial relation results with the groundtruth calculated by relative spatial relation computation unit.
 10. Thesystem for measuring tracker system accuracy according to claim 2,wherein the calibration pattern which is a pattern composed of ellipses.11. The system for measuring tracker system accuracy according to claim3, wherein the calibration pattern which is a pattern composed ofellipses.
 12. The system for measuring tracker system accuracy accordingto claim 4, wherein the calibration pattern which is a pattern composedof ellipses.
 13. The system for measuring tracker system accuracyaccording to Claim 8, wherein the calibration pattern which is a patterncomposed of ellipses.
 14. The system for measuring tracker systemaccuracy according to claim 9, wherein the calibration pattern which isa pattern composed of ellipses.
 15. The system for measuring trackersystem accuracy as in claim 2, wherein the calibration pattern which ismounted on the tracked object with a known previously measured spatialrelation with the tracked object, measured by the CMM.
 16. The systemfor measuring tracker system accuracy as in claim 3, wherein thecalibration pattern which is mounted on the tracked object with a knownpreviously measured spatial relation with the tracked object, measuredby the CMM.
 17. The system for measuring tracker system accuracy as inclaim 4, wherein the calibration pattern which is mounted on the trackedobject with a known previously measured spatial relation with thetracked object, measured by the CMM.
 18. The system for measuringtracker system accuracy as in claim 5, wherein the calibration patternwhich is mounted on the tracked object with a known previously measuredspatial relation with the tracked object, measured by the CMM.
 19. Thesystem for measuring tracker system accuracy as in claim 8, wherein thecalibration pattern which is mounted on the tracked object with a knownpreviously measured spatial relation with the tracked object, measuredby the CMM.
 20. The system for measuring tracker system accuracy as inclaim 9, wherein the calibration pattern which is mounted on the trackedobject with a known previously measured spatial relation with thetracked object, measured by the CMM.
 21. The system for measuringtracker system accuracy as in claim 10, wherein the calibration patternwhich is mounted on the tracked object with a known previously measuredspatial relation with the tracked object, measured by the CMM.
 22. Thesystem for measuring tracker system accuracy as in claim 11, wherein thecalibration pattern which is mounted on the tracked object with a knownpreviously measured spatial relation with the tracked object, measuredby the CMM.
 23. The system for measuring tracker system accuracy as inclaim 12, wherein the calibration pattern which is mounted on thetracked object with a known previously measured spatial relation withthe tracked object, measured by the CMM.
 24. The system for measuringtracker system accuracy as in claim 13, wherein the calibration patternwhich is mounted on the tracked object with a known previously measuredspatial relation with the tracked object, measured by the CMM.
 25. Thesystem for measuring tracker system accuracy as in claim 14, wherein thecalibration pattern which is mounted on the tracked object with a knownpreviously measured spatial relation with the tracked object, measuredby the CMM.